Papers with contextualized word representations
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing word embeddings were static, requiring all senses of a polysemous word to share the same representation. |
| Approach: | They found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model. |
| Outcome: | The results show that the representations of all words are not isotropic in any layer of the contextualizing model. |
Game Theory Meets Embeddings: a Unified Framework for Word Sense Disambiguation (D19-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is an open problem in Natural Language Processing (NLP). |
| Approach: | They propose a game-theoretic model that embeds ambiguous words as players of a non cooperative game and their senses as strategies that the players can select in order to play the games. |
| Outcome: | The proposed model performs well on standard benchmarks and different tests on standard datasets. |
Improving Word Sense Induction through Adversarial Forgetting of Morphosyntactic Information (2024.starsem-1)
Copied to clipboard
| Challenge: | Contextualized word representations from pre-trained language models encode more information than is necessary for the identification of word senses and some of this information affect performance negatively in unsupervised settings. |
| Approach: | They propose to use a framework to erase specific information from pre-trained word models and create feature-invariant representations that are invariant to these ‘nuisance features’. |
| Outcome: | The proposed framework erases information from the representations of pre-trained language models, thereby creating feature-invariant representations. |
Disambiguating Emotional Connotations of Words Using Contextualized Word Representations (2024.starsem-1)
Copied to clipboard
| Challenge: | BERT, RoBERTa, XLNet, and GPT-2 models effectively discern emotional connotations of words, demonstrating superior performance and greater resilience against biases. |
| Approach: | They propose to use contextualized word representations to examine how words can be used to distinguish emotional connotations across contexts. |
| Outcome: | The proposed models show that they can distinguish emotional connotations of words in different contexts. |
What Drives the Use of Metaphorical Language? Negative Insights from Abstractness, Affect, Discourse Coherence and Contextualized Word Representations (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing models of discourse coherence, abstractness and affect are not sufficient to explain metaphorical vs. literal language choices. |
| Approach: | They build five models that predict the use of a metaphorical vs. literal expression in context . they use established cognitive and linguistic properties to define discourse coherence, abstractness and affect . |
| Outcome: | The proposed models show that they do not explain metaphorical vs. literal language choices . metaphors are "not just nice", but represent a "necessary" element of everyday thought and communication . |
SUPP.AI: finding evidence for supplement-drug interactions (2020.acl-demos)
Copied to clipboard
Lucy Lu Wang, Oyvind Tafjord, Arman Cohan, Sarthak Jain, Sam Skjonsberg, Carissa Schoenick, Nick Botner, Waleed Ammar
| Challenge: | Dietary supplements are used by a large portion of the population, but information on their pharmacologic interactions is incomplete. |
| Approach: | They propose an application to search evidence sentences extracted from the literature to identify supplement-drug interactions. |
| Outcome: | The proposed model extracts supplement information and identifies interactions using labeled DDI data. |
Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis (2022.findings-naacl)
Copied to clipboard
| Challenge: | PPCEME has a large set of function tags and is difficult to parse . authors present results for PPceME using a modified version of the Berkeley Neural Parser . |
| Approach: | They propose to use a modified version of the Berkeley Neural Parser to parse PPCEME using function tags. |
| Outcome: | The proposed parser will be used to parse Early English Books Online, a 1.5 billion word corpus. |
Contextualized Word Representations for Reading Comprehension (N18-2)
Copied to clipboard
| Challenge: | Reading comprehension (RC) is a high-level task in natural language understanding that requires reading a document and answering questions about its content. |
| Approach: | They propose to provide a standard neural network for reading a document and answering a question about its content. |
| Outcome: | The proposed model improves on the competitive SQuAD dataset by providing rich contextualized word representations and allowing it to choose between context-dependent and context-independent representations. |
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Neural relation extraction models capture linguistic and semantic properties of the input, a recent study shows. |
| Approach: | They introduce 14 probing tasks targeting linguistic properties relevant to RE . they add contextualized word representations to enhance probing performance . |
| Outcome: | The proposed models achieve state-of-the-art on two datasets, TACRED and SemEval 2010 Task 8 . they show that the models capture linguistic and semantic properties relevant to the downstream task . |
A Unified Span-Based Approach for Opinion Mining with Syntactic Constituents (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-grained opinion mining (OM) are based on span-based annotations, but they are not effective. |
| Approach: | They propose a unified span-based approach for the end-to-end OM setting using syntactic constituents and multi-task learning to integrate them into the proposed model. |
| Outcome: | The proposed approach achieves significant improvements over previous work on the MPQA 2.0 dataset and reduces the number of wrongly-predicted opinion expressions and roles. |
Bird’s Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach (2021.acl-long)
Copied to clipboard
| Challenge: | Recent work on analyzing contextualized text representations has focused on hand-designed probe models to understand how and to what extent do these representations encode a particular linguistic phenomenon. |
| Approach: | They propose a new information-theoretic probe, Bird’s Eye, which detects if and how representations encode the information in contextualized text representations. |
| Outcome: | The proposed method estimates the mutual information between the linguistic graph embedded in a continuous space and the contextualized word representations. |
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (N19-1)
Copied to clipboard
| Challenge: | Recent work has shown that contextualized word representations are a viable alternative to simple word prediction tasks. |
| Approach: | They propose to use subword units and characters to model morphology, syntax, and semantics instead of word embeddings. |
| Outcome: | The proposed representations are better for modeling syntax and more robust to noisy input. |
Unsupervised Mapping of Arguments of Deverbal Nouns to Their Corresponding Verbal Labels (2023.findings-acl)
Copied to clipboard
| Challenge: | Deverbal nouns are nominal forms of verbs used in English texts to describe events or actions . many NLP systems neglect to handle nominalized constructions, resulting in limited applications . |
| Approach: | They propose to map arguments of deverbal nouns to universal-dependency relations of verbal constructions . they propose to use the same labels as verbal cases to map the arguments . |
| Outcome: | The proposed approach maps arguments of nominalized nouns to the corresponding verbal constructions. |
Semi-supervised Domain Adaptation for Dependency Parsing (P19-1)
Copied to clipboard
| Challenge: | Currently, most studies on cross-domain parsing focus on unsupervised domain adaptation . however, unsupervised approaches make limited progress due to the intrinsic difficulty of both domain adaptation and parse. |
| Approach: | They propose a semi-supervised domain adaptation problem for Chinese dependency parsing by using newly-annotated large-scale domain-aware datasets. |
| Outcome: | The proposed method is more effective than direct corpus concatenation and multi-task learning. |
Entity-Centric Contextual Affective Analysis (P19-1)
Copied to clipboard
| Challenge: | Existing methods for analyzing people portrayals take an unsupervised approach, or rely on domain-specific knowledge. |
| Approach: | They show how contextualized word embeddings can be used to capture affect dimensions in portrayals of people. |
| Outcome: | The proposed method can capture affect dimensions in portrayals of men and women . it is biased towards training data, which limits its usefulness to in-domain analyses . |
Syntax-Aware Opinion Role Labeling with Dependency Graph Convolutional Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Opinion role labeling (ORL) is a fine-grained opinion analysis task . due to the scarcity of labeled data, ORL remains challenging for data-driven methods due to its complexity and complexity. |
| Approach: | They propose to integrate syntactic knowledge into ORL models by comparing and integrating different representations and using dependency graph convolutional networks to encode parser information at different processing levels. |
| Outcome: | The proposed model achieves 4.34 higher F1 score than the current state-of-the-art. |
Is anisotropy really the cause of BERT embeddings not being semantic? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to train contextual language models for NLP use a lightweight approach called bi-encoder, which takes two sentences as input, but does not perform well with vanilla pre-trained Transformers. |
| Approach: | They conduct a set of experiments to improve our understanding of the lack of semantic isometry in contextualized word representations in BERT. |
| Outcome: | The proposed approach does not perform well with vanilla pre-trained Transformers. |
Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Recent advances in deep neural network models have improved parsing performance on in-domain texts . however, the problem is to improve performance on out-of-domain text data when there is only a small-scale out-domain labeled data. |
| Approach: | They propose to use adversarial learning and fine-tuning BERT to improve contextualized word representations on out-of-domain texts. |
| Outcome: | The proposed models achieve consistent improvement and fine-tune BERT processes boost parsing accuracy by a large margin. |
Context Analysis for Pre-trained Masked Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models that learn contextualized word representations from a large un-annotated corpus have become a standard component for many downstream NLP tasks. |
| Approach: | They propose to use a masking and gradient approach to evaluate the impact of context on the word representation. |
| Outcome: | The proposed model architectures are architecture agnostic and gradient based. |
Improving Disfluency Detection by Self-Training a Self-Attentive Model (2020.acl-main)
Copied to clipboard
| Challenge: | Existing self-attentive parsers using contextualized word embeddings produce state-of-the-art results in joint parsing and disfluency detection. |
| Approach: | They propose to use contextualized word embeddings to train a neural model using unlabeled data to train parsers. |
| Outcome: | The proposed method produces state-of-the-art results in parsing and disfluency detection in speech transcripts. |
Verb Sense Clustering using Contextualized Word Representations for Semantic Frame Induction (2021.findings-acl)
Copied to clipboard
| Challenge: | Contextualized word representations are effective in many natural language processing tasks, but it remains unclear to what extent they can cover hand-coded semantic information such as semantic frames. |
| Approach: | They compare contextualized word representations with two English frame-semantic resources . they find that several contextualized representations are informative for semantic frame induction . |
| Outcome: | The proposed representations are useful in natural language processing tasks, but are not fully understood by the literature. |
Towards the Detection of a Semantic Gap in the Chain of Commonsense Knowledge Triples (2022.lrec-1)
Copied to clipboard
| Challenge: | a commonsense knowledge resource organizes common sense that is not necessarily correct all the time, but most people are expected to know or believe. |
| Approach: | They propose a machine learning-based approach to detect semantic gaps in a commonsense knowledge graph . they use a conceptNet dataset to test the validity of two adjacent triples . |
| Outcome: | The proposed approach detects a semantic gap in a commonsense knowledge graph . the proposed approach also provides insights into the effectiveness of sense embeddings . |
Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations (D19-1)
Copied to clipboard
| Challenge: | Contextualized word representations are effective in downstream tasks such as question answering, named entity recognition, and sentiment analysis. |
| Approach: | They propose to integrate pre-trained contextualized word representations into a neural network that captures the whole sentence and the word representation in the sentence. |
| Outcome: | The proposed approach outperforms the state-of-the-art approach that makes use of non-contextualized word embeddings on multiple benchmark WSD datasets. |
An Attentive Fine-Grained Entity Typing Model with Latent Type Representation (D19-1)
Copied to clipboard
| Challenge: | Existing fine-grained entity typing models are criticized for label independence assumption . |
| Approach: | They propose a fine-grained entity typing model with a new attention mechanism and a hybrid type classifier to exploit type inter-dependency with latent type representation. |
| Outcome: | The proposed model significantly advances the state-of-the-art on fine-grained entity typing. |
A Joint Neural Model for Information Extraction with Global Features (2020.acl-main)
Copied to clipboard
| Challenge: | Existing joint neural models for Information Extraction use local task-specific classifiers to predict labels for individual instances. |
| Approach: | They propose a joint neural framework that extracts the optimal IE result as a graph from an input sentence. |
| Outcome: | The proposed model achieves new state-of-the-art on all subtasks and does not use any language-specific feature. |
Improving Neural Metaphor Detection with Visual Datasets (2020.lrec-1)
Copied to clipboard
| Challenge: | a new method for metaphor detection uses text from visual datasets to identify words . a metaphor is a complex interaction between two terms, creating an "implicationcomplex" |
| Approach: | They propose a technique for sampling text from visual datasets to create a visibility word embedding. |
| Outcome: | The proposed method improves on previous approaches that use more complex neural networks and richer linguistic features for verb classification. |